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ABSTRACT 

This project developed and tested an objective 
instructor assessment instrument to measure competencies of 
instructors who train Florida's child welfare and juvenile justice 
staff. Data from the literature review were used to identify and rank 
effective teaching competencies. A competency based instructor 
assessment scale was then developed using a behaviorally oriented 
rating scale approach in a format that would allow instructors to 
self-evaluate their classroom skills. The scale's criteria measure 
the knowledge, skills, and attitudes deemed necessary to conduct 
effective training. These criteria were extracted from the instructor 
competency categories of the International Board of Standards for 
Training, Performance, and Instruction. A formative review of the 
instrument was conducted by a five member committee that consisted of 
two senior trainers, two training administrators, and a training 
specialist. The draft instrument was then further refined and 
subjected to preliminary field testing. Results from the field test 
indicated that four raters had no difficulty using the instrument to 
rate two certified and two uncertified instructors. The assessment 
instrument was submitted to the Professional Development Centers 
headquarters and is being administered to all trainers during the 
certification process. Appendixes contain instructor competency 
categories, compet ency s urvey results, and the draft eva luat i on 
instrument . (Contains 1-8 references .) (JB) 
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THE DEVELOPMENT OF A COMPETENCY BASED 
INSTRUCTOR ASSESSMENT INSTRUMENT 

by 

Susann E. Rudasill 
August, i994 

The Department of Health and Rehabilitative Services (HRS) operates the 
Professional Development Centres (PDCs) to provide training for Florida's child 
welfare and juvenile justice staff. The purpose of this study was to develop an 
objective method to determine instructor competency in the classroom. The 
present instructor evaluation system does not adequately achieve that end. 

There were three research questions for this study. First, "Is there currently 
an accessible assessment instrument that wil! measure instructor competencies in 
the classroom?" Second, "Is it feasible to adapt an existing instrument or will it be 
necessary to design an assessment instrument that can be used to measure these 
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competencies?" Finally, "Can instructor classroom competencies be defined for 
the development of an assessment instrument using a Likert type scale 0 " 

Data from the literature review was used to rank order effective teaching 
competencies. A competency based instructor assessment instrument was then 
developed, using a behaviorally anchored rating scale approach. A formative 
review of the instrument was conducted by a five member committee that 
consisted of two senior trainers, two training administrators, and a training 
specialist. The draft instrument was then further refined and subjected to 
preliminary field testing 

Results from the field test indicated that four raters had no difficulty using the 
instrument to rate two certified and two uncertified instructors. The assessment 
instrument was submitted to PDC headquarters and is being administered to all 
trainers during the certification process. A recommendation was made for 
selection of a summative committee to evaluate the validity and reliability of the 
instrument. 
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Chapter 1 
INTRODUCTION 
Background and Significance 
The Department of Health and Rehabilitative Services (HRS) operates the 
Professional Development Centres (PDCs) to provide training for child welfare 
and juvenile justice staff who are employed by or under contract with the State of 
Florida. The Child Welfare and Juvenile Justice Standards and Training Councils 
are mandated by statute and were organized to advise HRS on the operation of the 
training system. The 1990 policy for certification of PDC instructors was 
developed and implemented in response to a recommendation from the joint 
councils. That policy was still in effect at the time this practicum report was 
proposed. 

Although the existing instructor certification policy emphasizes the need for 
ongoing staff development for instructors, it does not allow for measurement of 
the training skills that may need to be developed. The current policy does not 
define the knowledge, skills, and attitudes deemed necessary to conduct effective 
training. The present system that is used to measure instructor competency in the 
classroom relies on qualitative assessment by untrained observers. In 1993, a 
workgroup was formed to make recommendation for revisions to the certification 
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policy. The existing procedure for the evaluation of PDC instructors for their 
professional development and certification was reviewed. The workgroup 
recommended development of a quantitative and objective method to determine 
instructor competency in the classroom. 

The purpose of this practicum project was to determine if an instrument was 
available that would measure faculty competence in the classroom. The 
workgroup's stated criteria for selection or development of an instrument were as 
follows: 

• To identify and rank order effective teaching competencies. 

• To rate instructor's performance in the selected competencies. 

• To use the resulting assessment as a professional development planning 
tool. 

• To evaluate newly hired and experienced instructors for certification. 
It was also suggested that the assessment instrument should focus on common 
teaching techniques that would provide instructors with a frame of reference to 
self-evaluate their classroom skills. 

From an institutional perspective, several other benefits emerged during this 
developmental process. First, it became apparent that an objective assessment 
instrument would give supervisors a more impartial method of faculty evaluation. 
Second, the development process further defined the knowledge, skills, and 
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abilities necessary for instructors to conduct effective training. Third, it was 
thought that training participants would indirectly benefit from the use of the 
assessment instrument; in that instructor would ultimately improve their teaching 
techniques. Finally, this type of evaluation was considered an effective tool to 
encourage instructor self-assessment. 

Further review of the PDC's certification policy led to the conclusion that a 
standardized evaluation of new trainer \ is necessary— to assure fair selection of 
trainers for certification and to comply with equitable personnel procedures. The 
present system of instructor certification does not allow for committee review of 
candidates or self-assessment for professional development. Ultimately, an 
objective assessment instrument was determined to be a necessary element of an 
improved instructor certification process. 

The development of an instrument to be used as a professional development 
tool directly relates to the Human Resources Development seminar in that 
employee professional development was discussed as a sub-practice area of any 
organization (Carnevale et al., 1990). The seminar topics provided a rich source 
of materials from which to draw the information necessary to complete this 
practicum project. Participation in the seminar sessions and an assignment that 
required the implementation of an action plan, to further career development 
within the Professional Development Centre at Florida Atlantic University 
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(FAU/PDC), provided the impetus for this practicum project. That learning 
experience led to the discovery of a staff development problem. A preliminary 
analysis of that problem resulted in the development of the research questions that 
were addressed in this paper. 

Research Questions 
Three research questions emerged after further analysis of the need for an 
assessment instrument. First, "Is there currently an accessible assessment 
instrument that will measure juvenile justice and child welfare instructor 
competencies in the classroom?" Second, "Is it feasible to adapt an existing 
instrument or will it be necessary to design an assessment instrument that can be 
used to measure these competencies?" Finally, "Can instructor classroom 
competencies be defined for the development of an assessment instrument using a 
Likert type scale?" 

Definition of Terms 
For the purposes of this practicum report, selected terms were defined to add 
clarity to the project. The terms and their definitions follow alphabetically. 
Behaviorally anchored rating scales or BARS , is a method used to rank 
competencies that discriminate between behavior categories. 
C ontent validity of the instrument is defined as a judgment of the formative 
committee members, in their-selection of the competencies, that reflects how well 
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instructors manage the classroom and deliver the curriculum. 
Formative committee refers to the advisory workgroup, which consists of two 
senior trainers, two training administrators, and a training specialist, who are 
considered experts in training and development. 

Formative evaluation is defined as ongoing developmental review of the draft 
evaluation instrument. 

In structor competencies refer to the knowledge, skills, and attitudes deemed 
necessary for optimal instructor performance in the classroom. 
Instructor rating is a summation of average scores across items on the draft 
assessment instrument. 

PDC Raters refers to certified PDC senior instructors or training coaches that are 
selected to use the instructor assessment instrument to evaluate certified and 
uncertified instructors. 

Smile Sheets are the evaluation forms currently used by FAU/PDC to survey the 
participant's attitudes toward the entire training situation and the instructor. 
Summative evaluation refers to a final determination of the feasibility of field 
testing the assessment instrument for future use in the instructor certification 
program 
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Chapter 2 
REVIEW OF LITERATURE 
This literature review represents an expansion of the preliminary review 
conducted for the practicum proposal and was focused on recent literature in the 
field of Human Resources Development (HRD). This survey includes institutional 
research literature from several data bases, as well as texts and documents that 
pertain specifically to juvenile justice training and methods of instructor assessment 
and certification. Since the subject area under study is narrow, the search for an 
instructor assessment instrument was expanded to include teaching certification 
literature. 

Purpose of Certification 
According to the literature, training as a function can include a wide range of 
interventions intended to improve job performance. While there are many 
governmental organizations involved in credentialing or licensing, certification is 
usually the domain of private organizations by which professions are self- 
regulated. Gilley, Geis, and Seyfer (1987) define certification as a voluntary 
process that is used by an organization to measure and report on the degree of 
competence of individual practitioners. Unlike certification, licensing by 
governmental agencies is generally a credentialing process that is required by a 
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governing body. The American Psychological Association (APA, 1985) further 
defines this distinction and refers to licensure as the setting of minimum standards 
whereas certification sets standards above the minimum. 

For the purposes of this report, the professional certification definition was 
accepted as the standard that the assessment instrument would be selected or 
designed to meet. Gilley et al. (1987) further define certification and state that the 
process should: 

• Promote professional competencies. 

• Identify competencies unique to a profession. 

• Identify competencies for staff development. 

• Identify competent professionals and protect the public from 
incompetence. 

• Preempt governmental regulation or legislative action. 

Other researchers report that the certification process results in increased work 
satisfaction, credibility, and the establishment of clear expectations (Judd, 1988; 
Powers, 1992). There is however, a negative side to assessment and certification. 
John Kenny (1986) warns that the tools do not exist to assess the behavioral 
aspects of instructor's performance. 

The Ontario Society for Training and Development (OSTD, 1987), of which 
John Kenny is a member, published an instructor assessment and development 
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guide but issues certificates of achievement based on professional experience and 

education. The OSTD has no formal assessment of instructor performance ability. 

Gilley et al. (1987) outline several other problems associated with instructor 

assessment and claims that it is expensive and time consuming to develop and 

maintain reliable evaluation instruments. In a later study, Gilley and Galbraith r 

(1988) found that assessment of training performance in the classroom was present 

in 88.8% of the certifying organizations they surveyed. In these programs, 

certification is awarded to instructors who achieve a passing score for conducting 

training as measured by raters using a standardized evaluation instrument. The 

authors' survey also indicated that 97% of certifying organizations reported that 

instructor competencies were "at the heart" of their professional certification 

programs. 

Using the Certification Instrument 
Given the availability of an assessment instrument, Klingner and Nalbandian 
(1985) suggest caution when selecting raters. They claim that supervisory ratings 
are easily biased, self-ratings have received mixed research support, and peer 
ratings alone have proven acceptable in reliability and validity. These caveats led 
to further review of the literature to search for ways to achieve an acceptable level 
of validity in the development and administration of the proposed assessment 
instrument. Dwyer (1993) states that instructor assessment should involve the 
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instructor and be included in normal curriculum preparation. This idea was 
expanded and instructors 1 ideas and suggestions were included during the early 
stages of this project. Dwyer's concept of instructor involvement also resulted in a 
plan for instructor self-assessment and will be discussed in the procedures section 
of this report. 

Although much of the research that addresses instructor certification klcr f 'fies 
the use of assessment instruments (Jacobs, 1992), all the instruments are job 
specific and not many are competency based. It became apparent that job 
competencies must relate to the specific requirements of the teaching venue and 
the specific program requirements. The search for an instructor assessment 
instrument that would meet the PDC professional development and certification 
criteria was fruitless. Seemingly relevant materials such as Competency Analysis 
for Trainers (OSTD, 1987) was somewhat helpful but did not contain an 
assessment instrument that would have reflected the unique requirements of the 
PDC instructors' job. It was decided that a competency based instrument would be 
designed. 

Developing a Competency Based Assessment Instrument 
The developmental task was then re-focused on a search for an instiuctor 
competency model from which an instrument could be created. The exhaustive 
search for assessment instruments was helpful [a that it led to the discovery of 



ERLC 



17 



17 

several developmental models. Further review of the literature resulted in the 
examination of the International Board of Standards materials that contained a 
guide for the development of a competency based instructor assessment instalment 
(IBSTPI, 1992). !.n their book Analysis , the authors provide several systematic 
methods to develop behavioral assessment scales (Mills, Pace, & Peterson, 1989). 
After thorough review of the IBSTPI instructor competencies and a search for an 
appropriate instructor rating scale, it was decided to use a behavioral rating scale 
approach (Klingner & Nalbandian, 1985; Mills et al., 1989). The literature 
indicates that there are several advantages to using behaviorally anchored rating 
scales (BARS) as an alternative to conventional rating scales (Bennett & Ward, 
1993; Raymond & Houston, 1990; Wolf, 1993; Zemke & Kramlinger, 1987). 

Summary 

In sum, this literature review resulted in the illumination of a seldom- 
addressed problem in human resource development; that is, the importance of 
determining how instructor performance in the classroom can be equitably 
assessed. Although studies that specifically address the development of an 
assessment instrument were not found, research regarding instructor assessment 
and certification in other types of organizations provided several useful models. 
This comprehensive review also led to the discovery of a competency based 
approach that was used to develop and test an instructor assessment instrument. 
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Chapter 3 

METHODOLOGY AND PROCEDURES 
Product Design 

Several procedures were used to complete this development process. First, 
the literature review was expanded to determine if an instructor assessment 
instrument was available that could be customized to fit PDC needs. Second, 
international guidelines for the development of an instructor assessment instrument 
were reviewed to determine the feasibility of adapting that development model 
(IBSTPI, 1993). Third, a task analysis was performed to determine the skills, 
knowledge, and abilities, thought necessary to conduct successful training. Finally, 
since an appropriate instrument was not found, a formative committee, consisting 
of four senior trainers, and a training administrator, was selected to oversee the 
development process. This committee served as an advisory workgroup and 
monitored the development of the instructor assessment instrument. 

The 38 IBSTPI categories of competencies and performances were reviewed, 
by the formative committee, to determine which were most applicable to juvenile 
justice and child welfare instructors' performance in the classroom (Appendix A). 
This was accomplished by assigning several of the categories to each member of 
the formative committee according to their particular area of expertise. 
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Committee members then met with their center's PDC instructors and reviewed 
their assigned categories to define and validate the competencies and 
performances. Each sub-committee eliminated those functions that were deemed 
not applicable to their job. Each group then developed a list of the duties and 
tasks that applied to each of the remaining categories, focusing on exemplary 
performance for each competency. 

The formative committee reconvened the following month to compare the 
preliminary lists developed by each of their workgroups. After four half-day 
conference calls, the formative committee developed a combined list of the 20 
remaining competencies and performances that were deemed appropriate for 
further review. The committee then agreed to develop a survey to be submitted to 
PDC instructors and their supervisors for further validation. Another purpose of 
this survey was to verify the importance of each competency relative to the 
instructor's perceived ability in that skill. This comparison was accomplished by 
designing the survey to allow respondents to rate the importance of each 
competency as well as their perceived ability to perform each skill. 

To simplify rank ordering for the survey participants, two lists were prepared 
with each competency paired with every other competency. One list, titled 
Importance, instructed the respondent to select which of the paired competencies 
they felt was their strongest. The other list, titled Ability 9 asked the respondent to 
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select the competency in which they felt most accomplished. The results were then 
compiled using a self-assessment method for rank ordering developed by OSTD 
(1987). 

The survey results were then computed and the twenty competencies were 
ranked by their importance and the self-perceived ability of the survey respondents 
(Appendix B). The formative committee then reviewed the comparison to 
determine perceived performance deficits. The competencies that were rated high 
in importance and low in ability were considered critical ibr measuring instructor's 
professional development needs and were included in the assessment instrument. 
The survey information was also used to further validate the competency 
statements; that is, to determine if all respondents understood the statements and 
were able to relate them to classroom behavior. 

Once rank ordered, the competency statements were again reviewed by the 
formative committee and further defined for inclusion in the draft assessment 
instrument. The competency statements were then operationalized using a 
behaviorally anchored rating scale (BARS) approach (Mills et al., p. 85). The 
BARS approach was considered more applicable than use of a traditional likert 
scale for the development of this product. There were several reasons for this 
decision. First, this method discriminates between behavior categories that in turn 
reflect attitude, values, abilities, knowledge, and skills that increase or limit 
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instructor performance. Second, the specific performance is identified to help the 
rater make fewer subjective decisions. Finally, performance can be addressed by 
skill or task categories as opposed to a non-descriptive score or pass/fail grade. 
The BARS approach was revised to include the following: 

1. Using EBSTPI competencies, statements were developed that 
reflected successful and unsuccessful instructor performance. 

2. Statements were sorted into general categories describing the job 
behaviors and attached to those behaviors. 

3. Statements were further refined to measure differences in instructor 
performance and were sorted into a range of scaled values. 

4. Values were assigned to the scales for scoring and future certification 
cut-off points. 

The 20 competencies were subjected to the modified BARS approach and 
were used to develop the draft instructor assessment instrument (Appendix C). A 
field test was conducted by distributing the draft instrument to four PDC raters 
who individually rated four video presentations of training conducted by two 
certified and two uncertified juvenile justice instructors. Ratings were compared 
using simple descriptive statistics to establish the content validity of the instrument. 
Raters were also asked to comment, by item, as to the ease or difficulty in using 
the instrument. 
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Assumptions 

This practicum process was based on several assumptions. First, it was 
assumed that the formative committee members are content experts in the field of 
juvenile justice and child welfare training and curriculum development. Second, it 
was expected that the use of a behaviorally anchored rating scale approach would 
discriminate between behavior categories that in turn reflect attitudes, values, 
abilities, knowledge, and skills that enhance or limit instructor performance. 
Finally, it was anticipated that instructors and supervisors would view the 
assessment instrument as a professional development tool and not resist this type 
of professional evaluation. 

Limitations 

The ii rtructor assessment instrument is not intended to be used as a predictor 
of teaching effectiveness. Criteria for the development of the competencies that 
make up the instrument are appropriate for rating PDC juvenile justice and child 
wel.Ve instructors. Since these instructors train centralized instruction modules 
designed by the PDC's, the instrument may not be a valid measurement for other 
teaching situations. Determination of the reliability and validity of the draft 
assessment instrument can not be assumed as a result of the initial data collection 
and formative review. 
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Chapter 4 
RESULTS 

Review of the literature resulted in the decision to develop an assessment 
instrument that would measure instructor competency in the classroom. The 
literature also provided models for the methodology used to select and rank order 
the competencies. The criteria measured were based on the knowledge, skills, and 
attitudes, deemed necessary to conduct effective training. These criteria were 
extracted from the instructor competency categories found in the International 
Board of Standards for Training, Performance, and Instruction (IBSTPI, 1992). 

Initially, the competencies that were deemed related to optimal classroom 
performance were reduced in number to avoid redundancy. The remaining 
competencies were reviewed by the formative committee and rank ordered by each 
of the five members. The 19 selected competencies were the ~ separately rank 
ordered by importance and ability using the survey technique that was described in 
the procedures section of this report. 

Table 1 illustrates the selected competencies, that were ordered by the 
formative committee, to conform to the standardized PDC curriculum presentation 
format. 



Table 1 

Competency Categories 
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# Competency Title 



1 


Presentation' of Objectives 


2 


Participant Expectations 


3 


Exception Report 


4 


Professional Behavior 


5 


Eve Contact 


6 


Use of Voice 


7 


Distracting Trainer Behaviors & Mannerisms 


8 


Verbal Communication Skills 


9 


Questioning 


10 


Nonverbal Communication Skills 


11 


Reinforcement of Risk-Taking Behaviors 


12 


Examples, Anecdotes, Stories, Humor 


13 


Variation of Instructional Methods 


14 


Use of Instructional Aids 


15 


Facilitation of Learning Activities 


16 


Closure of Learning Activities 


17 


Handling Incorrect Responses and Statements 


18 


Handling Disruptive Behavior 


19 


Responding to Relevant Questions 


20 


Closure of the Session 
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The survey data was further analyzed to ascertain the performance gap between 
importance and ability in each competency. Simple descriptive statistics were then 
developed for later comparison to ratings of instructors using the assessment 
instrument (see Appendix B for raw data rankings). Figure 1 illustrates the 
contrast between instructor assessment of the importance of the competencies and 
their perceived ability in each category. 
Figure 1 

Competency Importance and Instructor Ability 




Competencies 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 

— I 1 \ \ \ 1 1 h— I 1 i I 1 1 1 1 l-H-H^H 1 



♦ Ability 



■ Importance 



The difference between instructor assessed ability and the ranked importance 
for each competency varied. Analysis of the data revealed nine competencies :wat 
were rated higher in perceived importance than in instructor ability. These 



competencies were identified for later weighting to determine how the assessment 
instrument will be scored. This information was also used to validate the inclusion 
of the selected competencies in the assessment instrument. Of the remaining 
competencies, eight were ranked higher in instructor ability than in perceived 
importance. Two competencies were rated equally in both categories. 
Competency nine was not ranked since it was added after the survey was 
completed. The eight competencies that were ranked highest in importance and 
lowest in ability are presented in table 2 in order of importance. 
Table 2 

Competencies Ranked Hig h er in Importance than Ability 



Rank Competency Title 



1 Handling Disruptive Behavior 

2 Professional Behavior 

3 Handling Incorrect Responses and Statements 

4 Reinforcement of Risk-Taking Behaviors 

5 Participant Expectations 

6 Presentation of Objectives 

7 Closure of the Session 

8 Distracting Trainer Behaviors and Mannerisms 

9 Exception Report 
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The 20 competencies were then subjected to the modified BARS approach 
that was described in the procedures section of this report. The resulting 
behavioral statements were used to develop the draft instructor assessment 
instrument (Appendix C). The PDC raters were asked to review the instrument 
before viewing the training presentation tapes. Because several of the performance 
scales were unclear to one of the raters, explanation notes were added to the 
instrument. Competency 18, handling disruptive behavior, was troublesome for 
two of the raters; therefore, a note was added to allow the possibility of no 
disruptive behavior taking place (see Appendix C, page 66). No response to this 
performance rating was tabulated as missing data and will not affect the overall 
average ratings of any instructor's performance. The draft assessment instrument 
was again revised to include a more detailed instruction section (see Appendix C, 
page 46). 

The four instructors who were rated were also given the opportunity to 
review the draft assessment instrument before the field test. This review was 
accomplished by allowing the instructors to use the assessment instrument to rate a 
taped video presentation of training conducted by a certified instructor to assure 
their understanding of the process. None of the instructors experienced any 
difficulty in understanding the competencies or rating the certified instructor. The 
two certified and two uncertified PDC instructors later individually self-assessed 
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their own video presentations of training. 

Draft instruments were then sent, with video tapes of instruction conducted by 
the two certified and two uncertified juvenile justice instructors, to the PDC raters. 
The raters were instructed to individually view each of the four-hour tapes and 
then rate each instructor. Before viewing tapes, the raters were asked to read and 
review the instrument. Raters were told that formative committee members were 
available to answer any questions regarding the use of the draft instrument. Raters 
were also told that the tapes should be privately viewed and the ratings kept 
confidential. 

The resulting rater scores by four raters of the four instructors were compiled 
and compared using simple descriptive statistics. The ensuing data analysis 
revealed that all raters were able to use the draft instrument to score all four video 
presentations of the two certified and two uncertified instructors. Feedback from 
each rater was solicited regarding the ease or difficulty of using the rating 
instrument. 

According to the raters, each review process took an average of six hours and 
none were conducted at one sitting. None of the raters experienced any difficulty 
in understanding and using the draft assessment instrument to rate each of the 
instructors. Table 3 provides a comparison of the certified and uncertified 
instructors' ratings by the PDC raters. 
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Table 3 

Mean Scores of Instructors by Raters 



PDC Raters 



Instructors 


#1 


#2 


#3 


#4 


#1 C 


3.44 


3.85 


3.36 


3.25 


#2C 


3.57 


3.99 


4.24 


3.75 


#3 UC 


2.51 


2.86 


3.08 


3.01 


#4UC 


2.74 


2.08 


3.23 


3.31 



Note . C = Certified UC=Un Certified. The scoring range was i (lowest) 
through 5 (highest). 

Examination of the instructors' mean sco^s by PDC raters, revealed that 
uncertified instructors' scores were lower, in all but one case, than certified 
instructors' scores. Uncertified instructor four was rated higher than certified 
instructor one by rater four. With this exception, the ratings of all certified 
instructors were higher than ratings for uncertified instructors. The ratings of all 
instructors appeared consistent across raters; that is, raters scored individual 
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instructors within one point on the rating scale. 

Grand means, by instructor and rater, were computed to further test the 
assumption that use of the assessment instrument allows raters to discriminate 
between inexperienced and experienced instructors. A comparison was then made 
to determine if rater means and grand means were lower for uncertified instructors 
than certified instructors. Figure 2 illustrates the comparison of certified and 
uncertified instructor means by rater. 
Figure 2 

Certified and Uncertified Instructor Means 



■ Rater #1 B Rater #2 □ Rater #3 B Rater #4 B Grand 

Mean 



Mean 
Scores 




Certified Uncertified 

As was expected, certified instructors' overall mean scores were consistently 
higher than uncertified instructors 1 scores. The results also indicated an absence of 
obvious rater bias; that is, individual raters did not show a pattern of high or low 
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ratings across instructors. To further evaluate the validity of the instrument, an 
item analysis was performed to determine if raters scored individual instructors 
consistently across the competencies. Raw data scores for each of the twenty 
competencies were compared, by viewing raw data, by rater for each instructor. 
This was a cumbersome process, and to depict those comparisons eight assessment 
items were selected at random to relate raters' scoring of one instructor. Figure 3 
represents the results of that test. 
Figure 3 

Competency Scores by Rater 

pi Rater #1 El Rater #2 □ Rater #3 H Rater #4 1 

5 
4 

Scores 3 
2 
1 

C8 C3 C12 C19 C17 C2 C5 C16 C13 CI 

The comparative inter-rater item scores shown in Figure 3, for one instructor, 
revealed relatively consistent scoring across raters. The ratings across 
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competencies varied as was expected— since the instructors would not logically be 
rated alike in each behavioral category. Identical tests run on the other three 
instructors produced similar results. The data was further analyzed to refine the 
instructor assessment instrument and answer the remaining research questions 
raised by this practicum process. To accomplish this, it was necessary to gain an 
understanding of how instructors self-assessed their performance using the draft 
instrument. A comparison between instructor self- assessment and rater assessment 
was made by comparing those mean scores. Figure 4 illustrates the results of that 
comparison. 
Figure 4 

Self- and Rater-Assessment Means 



■ Rater #1 



M Rater #2 



■ Rater #3 



a Rater #4 



□ Self Assmt. 



5 



Mean 
Scores 



4 - 
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3 - 
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Inst. #3 



Inst. #4 
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Examination of the self- and rater-assessment means reveal that instructors' 
self-assessed mean scores were generally consistent with the mean score 
assessments of all raters. Only one instructor self-assessed at a slightly higher 
score than one of the four raters. The remaining three instructors' self-assessments 
ranked one point or less than the assessments of their raters. These results 
remained consistent when instructors' self-assessments were compared by item. 

The draft instructor assessment instrument and raw data from which the 
preceding results were drawn will be kept in a computer data base and maintained 
by the FAU/PDC. The faculty and research staff can access these data for 
additional analysis and reports to HRS administrators. The draft assessment 
instrument and these results will also be made available to all instructors and raters 
who were involved in the further development and testing of the instrument. 



34 



34 



Chapter 5 

DISCUSSION, CONCLUSIONS, IMPLICATIONS 
AND RECOMMENDATIONS 
Discussion 

The purpose of this study was to determine the availability of an instructor 
assessment instrument that would identify the classroom competencies of 
experienced and inexperienced instructors employed by Florida's Professional 
Development Centres (PDCs). After an exhaustive review of the literature, it was 
determined that a relevant instrument, that would specifically reflect the needs of 
the PDC, was not available. The decision was then made to design an instrument 
that would serve as a frame of reference for instructors to self-evaluate their 
classroom skills. The procedures used for the purposes of this practicum project 
proved expedient to that end. 

During the development of the instrument, a survey was conducted to rank 
order the competencies that were selected from the International Board of 
Standards materials (IBSTPI, 1992). The data that resulted from that survey was 
retained to establish a set of core competencies from which the assessment 
instrument was developed. Further analysis of the data revealed additional 
information regarding the importance of certain competencies-specific to the 
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needs of the PDC's preferred instructor performance in the classroom. The survey 
resulted in the assessment of 20 rank ordered competencies that were used to 
determine an appropriately weighted scoring system for certification purposes. 

The behavioral rating scale approach (Klingner & Nalbandian, 1985; Mills et 
al., 1989) that was used to operationalize the competency categories, proved an 
efficient and reliable alternative to conventional rating scales. The resulting scales, 
that were presented to and edited by the formative committee, allowed selection of 
the most relevant and salient behavioral indicators for inclusion in the instrument. 
While this type of spontaneously developed instrument does not guarantee the 
validity and reliability of the instrument, its applications to the concerns of the 
specific organization's task are more readily defined. This assumption of relevance 
to the PDC instructor's task proved to be true; as is evidenced by the results of the 
instructor rankings. During the pilot test, all raters were able to rate the two 
certified and two uncertified instructors using the draft assessment instrument. It 
was decided that the ranked competencies proved relevant in determining 
instructor's classroom c kills. 

While the instrument was designed to allow for a competency based 
assessment, by rater, of instructor performance in the classroom, the preliminary 
field test supported an additional use for the assessment instrument. Specifically, 
the instrument proved effective as a self-assessment tool that permitted certified 
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and uncertified instructors to identify performance areas that require professional 
development. The self-assessment process, that was tested during this project, 
also allowed instructors to reflect on competency categories in which they felt 
proficient. The use of the assessment instrument for self-evaluation also enabled 
instructors to review behaviorally specific PDC expectations of their classroom 
performance. 

The instrument was also designed to be used in the certification process as a 
formal quantitative measure of instructor competency in the classroom. 
Instructors agreed that this method of instructor evaluation is more acceptable than 
the existing policy-that does not define the knowledge, skills, and abilities deemed 
necessary to conduct training. Instructors also commented that their involvement 
early in the certification process gave them an opportunity to contribute to the 
participative decision making process. This early instructor involvement was 
recommended by Dwyer (1993), as was explained in the literature review, and was 
incorporated in the procedures for this report. The ranked PDC performance 
objectives, according to all who reviewed the draft instrument, reflected i realistic 
view of the instructor's responsibilities in the classroom. 

Conclusions 

Before this study, there was no comprehensive analysis of PDC instructor 
competency in the classroom. Training participants' attitudes toward the entire 
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training situation sufficed as a measure of instructors' performance in the 
classroom. These observations were not recorded nor were they systematically 
reported. The assessment instrument that was developed as a result of this 
practicum project will replace that haphazard evaluation procedure. The results of 
the field test confirmed the assumption that inexperienced instructors would be 
rated lower than more experienced certified instructors. 

The identification of instructor competencies that are relevant to optimum 
performance in the PDC classroom will serve several purposes. First, standardized 
evaluation of new instructors will assure a minimum level of competence in the 
classroom. Second, new instructors will have the opportunity to view video tapes 
of experienced, certified instructors presenting training before taping their 
certification application presentation. Third, instructors applying for certification 
can self-assess their classroom skills based on a behaviorally anchored rating scale. 
Fourth, instructors will be rated by experienced raters using a standardized 
instrument. Finally, the instrument criteria, that are used for instructor assessment, 
can be used as minimum standards for hiring new instructors. 

The results indicate a general feeling of satisfaction with using the assessment 
instrument for professional development and the certification process. There was a 
general climate of dissatisfaction relayed regarding the previous unreliable 
instructor assessment process. Instructors want more information regarding 
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certification requirements, classroom skill development, and professional 
development strategies. Administrators indicated their appreciation of a more 
objective standard for the recruitment, hiring, and professional development of 
instructors. The successful administration of the competency survey indicated that 
instructors are willing to be assessed and to self-assess without fear of reprisal. 
This willingness to participate can be developed to create an ongoing training 
process for instructors. 

Implications 

With the implementation of a trainer assessment instrument and identification 
of trainer competencies that need improvement, training resources can be more 
appropriately allocated. A standardized evaluation of new trainers is also 
necessary to assure fair selection of trainers for certification and to ensure 
compliance with equitable personnel procedures. Continued piloting of the 
certification instrument will also encourage instructors to use the process as a 
professional development opportunity and seek ways to improve their skills. 
Ultimately, the assessment instrument can become a key element of an improved 
trainer certification process. The new assessment process provides a thorough 
analysis of instructor competency areas and identifies potential problem areas. 
This method of data gathering is effective, relatively inexpensive, and easily 
duplicated. 



The instructor ratings obtained as a result of the development and field testing 
of the draft assessment instrument provides a baseline data base from which the 
PDC system can conduct further research. Instructors will then have the option to 
work with senior instructors (coaches) to design an individualized plan, that is 
based on their competency assessment, for their professional development. The 
instructor assessment instrument can also be used as a professional development 
tool for veteran instructors as well as new instructors who wish to be considered 
for first time certification. 

The survey instrument also provided information that may be used to guide 
administrators planning future certification policy. Personnel administrators may 
want to consider looking at the allocation of resources when implementing new 
professional development programs. Involving employees in these decisions would 
certainly improve the method used to disseminate finite professional development 
resources. 

Recommendations for the Improvement of Practice 
The top rated competencies, that were also rated lower in perceived ability, 
indicate areas of concern for professional development. This information allowed 
a more concise statement of instructor competency. In every case, the 
performance deficit competencies can be addressed with specific skill building 
tools. All the ranked competencies were attached to specific classroom skills that 
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can be developed using a variety of professional development tools. Instructors 
who ranked above average in these critical competencies can serve as mentors for 
those instructors who rank below average. The video presentations also allow for 
the possibility of developing training tapes for the instructors. 

The process used to complete this study can also be used to determine 
employee attitudes, perceptions, expectations, and satisfaction with their 
organization's certification and professional development strategy. This 
recognition of employee, self, peer and senior certified instructor assessment, can 
be used to lessen the subjective nature of instructor evaluation. The assessment 
instrument can also be used to create a career ladder for PDC instructors who 
exceed the minimum requirements for certification and wish to achieve the rank of 
senior instructor. 

The assessment instrument will provide information that administrators may 
review before planning changes in the certification policy. Training administrators 
may want to consider looking at the allocation of resources when implementing 
new instructor training programs. Specifically, curriculum development and 
instructional design strategies can be modified to include a more comprehensive 
scope of instructor skills. Several of the competencies that exemplify optimum 
instructor performance in the classroom can be built into instructor guides that are 
now included in all standardized instruction modules. The existing curriculum can 
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be further enhanced to give instructors an opportunity to develop those 
competencies that the survey respondents deemed high in importance and low in 
instructor ability. 

The specific concerns that were expressed by instructors as a result of the self- 
assessment portion of this study, should cause training administrators to examine 
their assessment and certification policy decisions. It is evident that instructors 
seek an expanded role in decision making and desire more professional 
development opportunities. Training administrators can facilitate this process by 
dedicating resources to establish ongoing instructor involvement in the assessment 
and certification procedure. 

Following the final formative review and initial field test, the assessment 
instrument was submitted to PDC administrators for final approval. The 
instrument, pending final validation, is being administered to all trainers during the 
certification and re-certification process. A summative committee, composed of 
one PDC centre director, one HRS training administrator, one senior instructor, 
and an outside test development consultant, has been selected to review the 
assessment instrument. This committee will then develop a plan to determine the 
validity, reliability, and feasibility of future use of the instrument in the instructor 
certification and re-certification process. 
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Appendix A 
I nstructor Competency Categories 

INTRODUCTION 

• Preservation of Objectives 

• Introductory Activity 

• Exception Report 
PARTICIPATION 

• Verbal Language to Encourage Participation 

• Elicitation of Anecdotes, Examples, Analogies 

• Stories, and/or Humor from Participants 

• Reinforcement of Risk Taking Behavior with Participants 

• Handling Incorrect Responses and Statements of Participants 

• Frequency of Eye Contact 

• Distribution of Eye Contact 

• Non- Verbal Language to Encourage Participation 
INSTRUCTION 

• Volume and Clarity of Voice 

• Variation of Voice Tone 

• Variation of Instructional Methods 

• Handling Instructional Materials 
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Appendix A 
Instructor Competency Categories (Cont.) 
Non-verbal Language to Emphasize Content 
Distracting Behaviors and Mannerisms 
Management of Props and Audio/Visual Aids 
Anecdotes, Examples, Analogies, Stories, and/or Humor 
Application of Content to Job 
Closure on Learning Activities 
Referencing the Objectives 
Maintaining Participant Interest 
Time Management of the Session 
Managing Interaction 
Comfort in the Facilitator Role 
Explanation of Participant Roles 
Social Skills 

Ethical and Moral Behavior of the Instructor 
Instructor Errors 

Providing Requested Relevant Information to Participants 
Handling Disruptive Behavior among Participants 
Handling General Disruptive and Uncooperative Behavior 
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Appendix A 
Instructor Competency Categories (Cont.) 

• Management of Ethical and Moral Behavior of Participants 

• Leadership 
OVERALL CLOSURE 

• Meeting the Objectives of the Course 

• Closure of the Session 
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Appendix B 

Competencies Survey Results Ranked by Importance 
# Title Importance Ability 



1 8 Handling Disruptive Behavior 


1 


17 


4 Professional Behavior 


2 


9 


17 Handling Incorrect Responses and statements 


3 


5 


1 5 Facilitation of Learning Activities 


4 


4 


19 Responding to Relevant Questions 


5 


1 


12 Variation of Instructional Methods 


6 


3 


1 1 Reinforcement of Risk-Taking Behaviors 


7 


8 


2 Participant Expectations 


8 


13 


1 Presentation of Objectives 


9 


14 


8 Verbal Communication Skills 


10 


6 


20 Closure of the Session 


11 


18 


16 Closure of Learning Activities 


12 


12 


14 Use of Instructional Aids 


13 


7 


12 Examples, Anecdotes, Stories, Humor 


14 


2 


7 Distracting Trainer Behaviors & Mannerisms 


15 


16 


6 Use of Voice 


16 


11 


3 Exception Report 


17 


19 


10 Nonverbal Communication Skills 


18 


15 


5 Eye Contact 


19 


10 



9 Questioning— this competency added after the survey was completed 
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Appendix C 
Draft Evaluation Instrument 

THE INSTRUCTOR ASSESSMENT INSTRUMENT 

INSTRUCTIONS: 

This assessment instrument was designed specifically to evaluate instructors 
delivering the designated portions of training from a 32-hour pre-service module 
for the Professional Development System (PDC). This instrument may not be 
valid for use in other training situations. 

This instrument consists of nineteen behavioral rating scales that are to be used to 
evaluate instructor performance in the classroom. Raters should review the entire 
instrument before observing the instructor. The instrument should be completed 
after the rater has observed the entire four hours of training. Raters should select 
the rating on each scale that most closely resembles the behavior observed. A tally 
sheet is provided in this section that may be used to record the scores. 

Raters should read each item in each scale and all explanatory notes carefully when 
completing the evaluation. Variables that change include frequency, behavior, and 
level of involvement of participants. Usually, only one variable changes from one 
item to the next. However, more that one variable may change within a rating 
scale. 

Each rater should complete the evaluation independently. Other raters should not 
be consulted during the process. Common terms in the instrument include: 

Trainer = Instructor, Facilitator 
Participant = Trainee, Learner, Student 
Session = Course, Module, Lesson 

A number of scales consider frequency of behavior. Frequencies are defined as: 

Never 

Occasionally (sometimes) 
Frequently (a lot) 
Always 
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Appendix C 
Draft Evaluation Instalment (Cont.) 

A number of scales consider relative numbers (of participants, objectives, etc.). 
The continuum for relative numbers is as follows: 
None 

Some (a few) 
Most (many) 
All 

No percentages or numbers are used to determine the rating with either 
continuum. There are only four discrete categories in each continuum. Raters 
should be able to make choices without counting. 

There is more than one correct way for instructors to handle many situations. 
Employing a variety of skills is generally preferable to using only one approach for 
a particular situation. Therefore, the scales often provide criteria for behaviors 
rather than precise behaviors. 

Trainers can meet many of the standards either directly or indirectly through group 
facilitation. Either method is usually satisfactory for ratings. If the statement says 
"the trainer" and the participants actually accomplish the task, it meets the criteria 
for the rating. 

It may be possible for a trainer to meet the criteria for more than one rating in the 
same scale because the session takes place over time. If this occurs, the rater must 
select a rating based on the relative impact or frequency of the behaviors observed. 

Raters should not score an item if it is not applicable. There may be some cases 
where there was no opportunity to accomplish a particular performance. For 
example, there will be no opportunity for a trainer to handle an incorrect statement 
if participants do not make any. When no score is entered, it will be entered as 
missing data. It will have no effect on the overall average or mean score. 
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Draft Evaluation Instalment (Cont.) 
THE RATING SCALES 

Page 



1. PRESENTATION OF OBJECTIVES 52 

2. PARTICIPANT EXPECTATIONS 53 

3. EXCEPTION REPORT 54 

4. PROFESSIONAL BEHAVIOR 55 

5. EYE CONTACT 56 

6. VOICE 57 

7. DISTRACTING INSTRUCTOR BEHAVIORS AND MANNERISMS 58 

8. VERBAL COMMUNICATION SKILLS 59 

9. QUESTIONING TECHNIQUES 60 

10. NON-VERBAL COMMUNICATION SKILL 61 

11. RISK-TAKING BEHAVIORS OF PARTICIPANTS 62 

12. EXAMPLES, ANECDOTES, ANALOGIES, AND HUMOR 63 

13. INSTRUCTIONAL METHODS 64 

14. INSTRUCTIONAL AIDS 65 

15. FACILITATION OF LEARNING ACTIVITIES 6 6 

16. CLOSURE OF LEARNING ACTIVITIES 67 

17. INCORRECT PARTICIPANT RESPONSES ABOUT CONTENT .. 68 

18. DISRUPTIVE PARTICIPANT BEHAVIOR 69 

19. RELEVANT PARTICIPANT QUESTIONS 70 

20. CLOSURE OF THE SESSION 71 
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Draft Ev aluation Instalment (Cont.) 
1 

PRESENTATION OF OBJECTIVES 

5. The trainer presented ALL of the objectives at the beginning of the session 
ORALLY AND IN WRITING and EXPLAINED them. 

4. The trainer presented ALL of the objectives at the beginning of the session 
ORALLY AND IN WRITING. 

3. The trainer presented ALL of the objectives at the beginning of the session. 

2. The trainer presented SOME of the objectives at the beginning of the session. 

1 . The trainer DID NOT present any of the objectives at the beginning of the 
session. 




NOTE: "Related" means compared similarities and differences between 



expectations and objectives. 
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Draft Evaluation Instrument (Cont.) 



PARTICIPANT EXPECTATIONS 



5. The trainer provided the opportunity for ALL participants to communicate 
expectations of the session and RELATED them to the OBJECTIVES of the 



session. 



4. The trainer provided the opportunity for SOME participants to communicate 
expectations of the session and RELATED them to the OBJECTIVES of the 



session. 



3. The trainer provided the opportunity for ALL participants to communicate 
expectations of the session. 

2. The trainer provided the opportunity for SOME participants to communicate 
expectations. 



1 . NO opportunity was provided for participants to communicate expectations. 



NOTE: "Related" means compared similarities and differences between 
expectations and objectives. 
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Draft Evaluation Instalment (ConU 
3 

EXCEPTION REPORT 

5. The trainer INFORMED participants of the trainee exception report at the 
BEGINNING of the session, and THOROUGHLY EXPLAINED it's use or 
determined that the participants had a thorough understanding from a previous 
session. 

4. The trainer INFORMED participants of the trainee exception report at the 
BEGINNING of the session and provided SOME EXPLANATION of its use. 

3. The trainer INFORMED participants of the trainee exception report at the 
BEGINNING of the session. 

2. The trainer INFORMED participants of the trainee exception report DURING 
the session (after the first module had formally begun). 



1 . The trainer DID NOT INFORM participants of the trainee exception report. 




Draft Evaluation Instrument (Cont.) 



PROFESSIONAL BEHAVIOR 



5. The trainer MODELED PROFESSIONAL behavior in all interactions and 
ESTABLISHED the same STANDARDS for participants. 



4. The trainer MODELED PROFESSIONAL behavior in all interactions. 



3. The trainer MODELED PROFESSIONAL behavior. 



2. The trainer was OCCASIONALLY UNPROFESSIONAL. 



1 . The trainer's behavior was UNPROFESSIONAL. 



NOTE: PROFESSIONAL BEHAVIOR IS RESPECTFUL and COURTEOUS of 
all groups, subgroups, and individuals and adheres to ETHICAL and MORAL 
standards. Examples of unprofessional behavior include actions, comments, 
stories, and humor that promote illegal, dishonest, racist, or sexist attitudes or 
actions tl< i demean or express bias or prejudice against ethnic groups or 
individuals, including social service clients. 
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Draft Evaluation Instrumen (Cont.) 



EYE CONTACT 



5. The trainer FREQUENTLY made eye contact with MOST participants. 



4. The trainer OCCASIONALLY made eye contact with MOST participants. 



3 . The trainer FREQUENTLY made eye contact with SOME participants. 



2. The trainer OCCASIONALLY made eye contact with SOME participants. 



1 . The trainer FREQUENTLY AVOIDED eye contact. 



NOTE: Intentional use of the technique of avoiding eye contact to discourage 
inappropriate participation should not be considered when rating this item. 
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Draft Evaluation Instalment (Com.) 
6 

VOICE 

5. The trainer spoke LOUDLY and CLEARLY enough to be heard and 
understood by everyone in the room and FREQUENTLY VARIED vocal 
characteristics. 

4. The trainer spoke LOUDLY and CLEARLY enough to be heard and 

understood by everyone in the room and OCCASIONALLY VARIED vocal 
characteristics. 

3 . The trainer spoke LOUDLY and CLEARLY enough to be heard and 
understood by everyone in the room. 

2. The trainer FREQUENTLY DID NOT speak LOUDLY and CLEARLY 
enough to be heard and understood by everyone in the room. 

1 . The trainer could not be heard and spoke in a MONOTONE voice. 
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Draft Evaluation Instalment fCont.) 
7 

DISTRACTING TRAINER BEHAVIORS AND MANNERISMS 

5. The trainer DID NOT display distracting behaviors and mannerisms. 

4. The trainer displayed MINIMALLY distracting behaviors and mannerisms. 

3. The trainer OCCASIONALLY displayed distracting behaviors and 
mannerisms. 

2. The trainer FREQUENTLY displayed distracting behaviors and mannerisms. 
1 . The trainer displayed EXTREMELY distracting behaviors and mannerisms. 




NOTE: Examples of DISTRACTING BEHAVIORS and MANNERISMS can 
include; aimless gestures and movements, jingling items in pockets, turning away 
from participants for long periods, and excessive use of verbal fillers such as M er," 
"urn," and "you know." Isolated incidents of these examples should not be 



considered distracting. 
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Draft Evaluation Instrument (Cont.) 
8 

VERBAL COMMUNICATION SKILLS 

5. The trainer used verbal communication kills to emphasize important points, 
demonstrate interest (of the trainer), and GENERATED ACTIVE 
PARTICIPATION. 

4. The trainer used verbal communication skills to emphasize important points, 
demonstrate interest, and GENERATED PARTICIPATION. 

3. The trainer FREQUENTLY used verbal communication skills to emphasize 
important points, demonstrate interest, but RARELY GENERATED 
PARTICIPATION. 

2. The trainer OCCASIONALLY used verbal communication skills to emphasize 
important points, and demonstrate interest, but DID NOT GENERATE 
PARTICIPATION. 



1 . The trainer DID NOT use verbal communication skills to emphasize important 
points or demonstrate interest. 
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Draft Evaluation Instrument (ConO 
9 

QUESTIONING TECHNIQUES 

5. The trainer FREQUENTLY asked OPEN-ENDED or CLOSED-ENDED 
QUESTIONS which related to the objectives and encouraged participation. 

4. The trainer OCCASIONALLY asked OPEN-ENDED or CLOSED-ENDED 
QUESTIONS which related to the objectives and encouraged participation. 

3 . The trainer OCCASIONALLY asked OPEN-ENDED or CLOSED-ENDED 
QUESTIONS which related to the objectives. 

2. The trainer asked OPEN-ENDED and CLOSED-ENDED QUESTIONS 
which were NOT related to the objectives. 

1 . The trainer DID NOT ask OPEN-ENDED or CLOSED-ENDED 
QUESTIONS. 
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Draft Evaluation Instrument (Cont.) 
10 

NON-VERBAL COMMUNICATION SKILLS 

5. The trainer used non-verbal communication skills to emphasize important 
points, demonstrate interest (of the trainer), and GENERATED ACTIVE 
PARTICIPATION. 

4. The trainer used non-verbal communication skills to emphasize important 
points, demonstrate interest and GENERATED PARTICIPATION. 

3 . The trainer FREQUENTLY used non-verbal communication skills to 
emphasize important points, and demonstrate interest but RARELY 
GENERATED PARTICIPATION. 

2. The trainer OCCASIONALLY used non-verbal communication skills to 
emphasize important points, and demonstrate interest but DID NOT 
GENERATE PARTICIPATION. 

1 . The trainer DID NOT use non-verbal communication skills to emphasize 



important points or demonstrate interest. 




NOTE: Examples of NON-VERBAL COMMUNICATION SKILLS include 
facial expressions, gestures, body movement, and position. Verbal and non-verbal 
communication are not mutually exclusive. Both can occur simultaneously. 
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RISK-TAKING BEHAVIORS OF PARTICIPANTS 



5 . The trainer ENCOURAGED and ACKNOWLEDGED risk-taking behaviors. 



4. The trainer ENCOURAGED risk-taking behaviors. 



3 . The trainer ACKNOWLEDGED risk-taking behaviors. 



2. The trainer DID NOT ACKNOWLEDGE risk-taking behaviors. 



1 . The trainer DISCOURAGED risk taking behaviors. 
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USE OF EXAMPLES, ANECDOTES, STORIES, 
ANALOGIES, AND/OR HUMOR 

The trainer FREQUENTLY ELICITED AND PROVIDED examples, 
anecdotes, stories, analogies, and/or humor. 

The trainer OCCASIONALLY ELICITED AND PROVIDED examples, 
anecdotes, stories, analogies, and/or humor. 

The trainer PROVIDED OR ELICITED examples, anecdotes, stories, 
analogies, and/or humor. 

The trainer DID NOT PROVIDE OR ELICIT examples, anecdotes, stories, 
analogies, and/or humor. 

The trainer PROVIDED IRRELEVANT examples, anecdotes, stories, 
analogies, and/or humor. 
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INSTRUCTIONAL METHODS 



5. The trainer FREQUENTLY VARIED INTERACTIVE instructional methods. 



4. The trainer FREQUENTLY VARIED instructional methods. 



3 . The trainer OCCASIONALLY VARIED instructional methods. 



2. The trainer relied PRIMARILY on ONE instructional method (e.g., lecture). 



1 . The trainer relied EXCLUSIVELY on ONE instructional method. 



NOTE: Examples of INSTRUCTIONAL METHODS include; small group 
interaction, games, role plays, guided discussions, simulations, structured 
exercises, lectures, flip charts, overheads, videos, case studies, and debriefing 
discuss'ons. 
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USE OF INSTRUCTIONAL AIDS 

5. The trainer used a VARIETY of instructional aids to DEMONSTRATE 
CONTENT and INVOLVE PARTICIPANTS. 

4. The trainer used a VARIETY of instructional aids to DEMONSTRATE 
CONTENT. 



3 . The trainer OCCASIONALLY used instructional aids to DEMONSTRATE 
CONTENT. 



2. The trainer's use of instructional aids was DISTRACTING and interfered with 
involvement of the participants. 



1 . The trainer used instructional aids which were IRRELEVANT and 



DISTRACTED from the content. 




NOTE: Examples of INSTRUCTIONAL AIDS include; notes, handouts, case 
studies, flipcharts, chalk boards, VCRs, and other audio/visual aids. Examples of 
DISTRACTING and interfering use of instructional aids include excessive reading, 
inadequate number of handouts, handouts that are out of order or missing, not 
having working markers, and not being able to operate equipment. 
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FACILITATION OF LEARNING ACTIVITIES 

5 . The trainer THOROUGHLY EXPLAINED participant roles, ASKED for 
FEEDBACK, and MONITORED the learning activities. 

4. The trainer THOROUGHLY EXPLAINED participant roles and 
MONITORED the learning activities. 

3 The trainer EXPLAINED participant roles and MONITORED the learning 
activities. 

2. The trainer EXPLAINED participant roles for the learning activities. 

1 . The trainer INADEQUATELY EXPLAINED participant roles for the 
learning activities. 
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CLOSURE OF LEARNING ACTIVITIES 



5. The trainer involved MOST participants in DISCUSSION AND REVIEW of 
major points and objectives at the conclusion of learning activities. 



4. The trainer involved SOME participants in DISCUSSION AND REVIEW of 
major points and objectives at the conclusion of learning activities. 

3. The trainer REVIEWED major points at the conclusion of activities. 



2. The trainer OCCASIONALLY REVIEWED major points at the conclusion of 
SOME activities. 



1 . The trainer DID NOT REVIEW major points or objectives at the conclusion 
of activities. 
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INCORRECT PARTICIPANT RESPONSES OR STATEMENTS 
CONCERNING CRITICAL CONTENT 

5. The trainer ALWAYS provided the correct response and provided POSITIVE 
reinforcement for participation. 

4. The trainer ALWAYS provided the correct response and OCCASIONALLY 
provided POSITIVE reinforcement for participation. 

3. The trainer ALWAYS provided the correct response and WITHOUT 
providing POSITIVE or NEGATIVE reinforcement 

2. The trainer provided the correct response and provided NEGATIVE 
reinforcement for participation. 



1. The trainer IGNORED incorrect responses or statements. 




NOTE: The trainer can provide the correct response either directly or indirectly. 
Content refers strictly to the curriculum for the session. Examples of indirectly 
providing response can include; rephrasing, restructuring, repeating questions, 
using another participant to provide the correct response, or directing the 
participants to the correct response. 
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DISRUPTIVE PARTICIPANT BEHAVIORS 

5. The trainer INTERVENED in a CALM, INOFFENSIVE manner and used the 
disruptive behaviors as LEARNING EXPERIENCES. 

4. The trainer INTERVENED in a CALM, INOFFENSIVE manner and 
STOPPED the disruptive behaviors. 

3 . The trainer INTERVENED and STOPPED the disruptive behaviors. 

2. The trainer CONFRONTED and DID NOT STOP the disruptive behaviors. 

1 . The trainer IGNORED ALL disruptive behaviors. 



NOTE: Examples of DISRUPTIVE BEHAVIOR include: continually holding 
side conversations; refusing to participate in activities; reading the newspaper; 
interrupting, insulting, or laughing at others; or, other unprofessional behavior. IF 
NO DISRUPTIVE BEHAVIOR OCCURS, DO NOT RATE THIS 
PERFORMANCE. 
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CLOSURE OF THE SESSION 

The trainer INVOLVED MOST participants in a DISCUSSION OF THE 
OBJECTIVES and RELATED them to the JOB at the conclusion of the 
session. 

The trainer INVOLVED SOME participants in a DISCUSSION OF THE 
OBJECTIVES at the conclusion of the session. 

The trainer REVIEWED THE OBJECTIVES at the conclusion of the session. 

The trainer requested COMMENTS from participants at the conclusion of the 
session. 

The trainer closed the session WITHOUT REQUESTING COMMENTS or 
REVIEWING OBJECTIVES. 



